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Abstract1 

The  System  Software  Safety  Technical  Review  Panel  (SSSTRP)  is  tasked  with 
reviewing  the  software  safety  processes  and  practices  of  US  Navy  software-intensive  Gun 
System  acquisition  programs  from  the  early  stages  of  the  acquisition  process.  As  these 
systems  grow  in  complexity  and  as  Open  Architecture  (OA)  is  implemented,  the  acquisition  and 
demonstration  of  safe  software  is  becoming  a  more  challenging  task — often  resulting  in 
unexpected  safety  risks,  schedule  delays,  and  cost  overruns.  This  research  presents  an 
approach  to  mitigate  common  risks  in  this  domain  from  the  Program  Management  level.  This 
approach  focuses  on  analyzing  historical  weapon  system  SSSTRP  data  to  identify  trends  that 
could  lead  to  a  strategy  to  increase  software  safety  as  well  as  reduce  unexpected  findings  at  the 
SSSTRP.  This  research  effort  is  still  in  the  early  stages,  but  data  are  being  collected,  and 
progress  is  being  made.  The  goal  of  this  paper  is  to  increase  awareness  of  both  the  problem 
and  the  research  effort  that  is  attempting  to  mitigate  the  common  effects  felt  by  Program 
Managers. 

Background 

The  United  States  Navy  (USN)  formed  the  Weapon  Systems  Explosive  Safety  Review 
Board  (WSESRB)  in  1968  as  a  result  of  the  tragic  fire  onboard  USS  FORRESTAL  (CV  59).  The 


1  This  work  was  supported  in  part  by  ARO  under  project  number  P-45614-CI  and  in  part  by  NAVSEA 
under  project  number  09WX1 1455. 
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subsequent  investigation  recommended  an  independent  review  process  be  established.  The 
report  highlighted  the  need  to  ensure  explosives  safety  requirements  are  met  for  all  munitions 
introduced  to  the  Fleet.  WSESRB  members  participate  in  numerous  weapons  system  safety- 
related  meetings,  technical  reviews,  and  working  groups. 

The  WSESRB's  responsibility  is  to  review  the  overall  safety  aspects  of  each  weapon 
system,  explosive  system,  and  related  system  to  ensure  that  weapon  system  safety 
requirements  are  satisfied.  Having  assessed  the  degree  of  compliance  with  existing  criteria,  the 
WSESRB  provides  an  assessment  of  the  adequacy  of  the  safety  program  and  makes 
recommendations  on  the  advancement  of  the  item  to  the  next  stage  in  the  acquisition  cycle  to 
the  program  manager,  program  sponsor,  Chief  of  Naval  Operations  (CNO),  and  the  Milestone 
Decision  Authority  (MDA). 

At  the  discretion  of  the  WSESRB  Chairperson,  special  WSESRB  Technical  Review 
Panels  (TRPs)  may  be  established  to  review  specific  safety  aspects  requiring  special  expertise 
(e.g.,  ordnance-related  software  safety)  of  weapon  systems.  These  TRPs  are  scheduled  and 
led  by  an  appointed  TRP  Chairperson  and  have  at  least  two  other  designated  members.  Naval 
Systems  Commanders,  when  requested  by  the  WSESRB  Chairperson,  may  identify  a  member 
to  serve  on  TRPs.  These  members  are  familiar  with  the  responsibilities  of  their  Systems 
Commands  and  respective  program  requirements  and  have  expertise  in  the  applicable  area  of 
the  TRP.  Other  members  and  technical  advisors,  chosen  for  their  expertise,  are  appointed  at 
the  discretion  of  the  TRP  Chairperson.  Software  System  Safety  Technical  Review  Panel 
(SSSTRP)  is  one  of  these  special  WSERB  TRPs. 

Recommendations  made  by  TRPs  will  be  presented  to  the  Program  Office  and  the 
WSESRB  at  the  conclusion  of  the  TRP  meeting;  however,  they  do  not  become  official  until 
reviewed  and  endorsed  by  the  WSESRB.  The  WSESRB  may  accept,  modify,  or  reject  the 
recommendations  of  the  TRP.  The  results  of  the  WSESRB  action  on  the  TRP 
recommendations  will  be  provided  to  the  Program  Office. 

Naval  Surface  Warfare  Center  (NSWC)  Dahlgren  Division  acts  as  a  principal  activity  for 
system  safety  support  to  the  WSESRB,  as  well  as  chairing  the  SSSTRP  and  other  TRPs  as 
assigned.  These  assignments  include:  (1)  developing  and  recommending,  with  WSESRB 
approval,  TRP  review  criteria  and  related  data,  (2)  coordinating  meetings  of  the  SSSTRP  with 
members  and  program  offices,  (3)  assisting  the  program  office  in  tailoring  TRP  review  criteria 
for  each  type  of  program  and  current  program  phase,  (4)  identifying  qualified  technical  advisors 
to  participate  in  the  TRP,  and  with  the  WSESRB  chairperson’s  concurrence,  arranging  for  their 
participation,  (5)  scheduling  meetings  of  the  TRP  at  the  request  of  the  WSESRB  chairperson, 
and  (6)  providing  a  summary  report  of  the  findings  and  recommendations  of  the  SSSTRP  TRP 
to  the  WSESRB. 

The  SSSTRP's  primary  focus  is  to  investigate  whether  the  vendor's  software  engineering 
processes  properly  identify  and  address  the  risks  associated  with  the  implementation  of  their 
product.  The  following  list  represents  the  SSSTRP's  areas  of  focus  within  the  software 
development  process: 

•  Software  Development  Process  Essentials 
o  Software  Development  Plan 
o  Configuration  Control  Management 
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o  Requirements  Management 
o  Safety  Involvement 

■  Change  Boards 

■  Trouble  Reports 

■  Build  Reviews 

■  Test  &  Integration  Plans 

Vendors  are  required  to  submit  Technical  Data  Packages  (TDPs)  that  contain 
documentation  that  details  the  vendor's  quality  control  procedures  associated  with  the  following 
engineering  processes: 

•  Software  Engineering 

o  Software  Development  Plan 
o  Software  Architecture  Design 
o  Software  Interfaces 
o  Software  Detailed  Design 
o  Software  Testing  &  Integration  Plans 
o  Software  Verification  &  Validation  Plans 
o  Software  Build  Schedule 
o  Build  Milestones 
o  Build  Functionality 

•  Specialty  Engineering 

o  Configuration  Management 
o  Requirements  Management 

The  actual  components  of  the  TDP  are  made  up  of  the  project  documentation  submitted 
by  the  vendor.  The  vendor  submits  the  following  documents  as  the  TDP: 

•  System  Program 

o  System  Description 
o  Program  Organization 
o  Program  Schedule 
o  Concept  of  Operations 

•  Software  Program 
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o  Software  Development  Plan 
o  Software  Build  Plan 
o  Software  Configuration  Management 
o  Software  Requirements  Management 
o  Software  Change  Board  Control 

•  System  Safety  Program 

o  System  Safety  Program  Plan 

■  Integrated  Safety  Schedule 

■  Safety  Organization 

•  Roles  &  Responsibilities 

•  IPT  (Cross  Product  Team)  Interactions 

■  Software  Safety  Analyses  Descriptions 

■  Hazard  Tracking  System 

■  System  Safety  Working  Group  (SSWG) 

•  Current  Safety  Status 

o  Preliminary  Hazard  List  (PHL) 

The  vendor's  responsibility  during  the  SSSTRP  presentation  is  to  identify  the  risks 
associated  with  its  products  relative  to  all  stages  of  the  software  lifecycle.  The  vendor  is  also 
required  to  present  the  associated  mitigation  strategy  for  each  risk.  Software  risks  can  be  found 
via  the  following  analysis  techniques: 

•  Functional  Hazard  Analysis  (FHA) 

•  Software  Requirements  Analysis  (SRA) 

•  Preliminary  Design  Analysis  (PDA) 

•  Detailed  Design  Analysis  (DDA) 

•  COTS  Analysis 

•  Code  Analysis  (CA) 

•  Software  Test  Results  Analysis  (STRA) 


Problem  Statement 

Current  and  active  US  Navy  gun  system  acquisition  programs  are  more  complex  than 
their  predecessors.  The  systems  being  acquired  are  much  more  software-intensive  and  present 
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a  much  greater  challenge  with  respect  to  understanding  and  mitigating  the  risks  associated  with 
Navy  gun  safety.  Navy  gun  systems  are  much  more  software-intensive  than  previous 
generations  due  to  the  increasingly  complex  requirements  for  centralized  command  and  control 
(C2).  In  order  to  adapt  to  new  technologies,  the  Navy  has  engaged  the  Open  Architecture  (OA) 
concept.  Although  the  OA  approach  is  much  more  flexible,  there  are  inherent  risks  associated 
with  it.  The  Navy  acquisition  community,  specifically  the  SSSSTRP  process,  needs  to  adapt  to 
this  new  OA  environment  and  standardize  the  SSSTRP  processes. 

Significant  effort  is  required  to  put  together  a  TDP  for  SSSTRP  review.  The  SSSTRP 
panel  finds  issues  in  the  vast  majority  of  systems  that  come  before  it.  Such  findings  need  to  be 
addressed  prior  to  WSESRB  concurrence  on  the  overall  program.  Typically,  every  system 
going  in  front  of  the  WSESRB  and  SSSTRP  is  handled  on  a  strictly  individual  basis  by  its 
respective  vendor  and  Program  Managers  (PMs).  PMs  ideally  want  their  programs  to  be  well 
prepared  for  the  SSSTRP  so  there  are  no  unexpected  surprises  resulting  in  safety  hazards, 
schedule  delays  and/or  cost  overruns.  However,  Navy  gun  system  PMs  do  not  have  a  good 
handle  on  what  the  trends  are  in  the  findings  from  system  to  system,  and  there  is  always  a 
desire  to  reduce  the  programmatic  risk  involved  in  passing  the  reviews. 

If  commonalities  or  trends  in  SSSTRP  findings  were  identified,  they  could  be  analyzed — 
leading  to  a  better  programmatic  strategy  to  generate  safe  systems.  This  would  also  result  in  a 
minimal  number  of  hurdles  during  the  SSSTRP  review,  and  the  end  result  would  be  beneficial  to 
all  parties  involved. 

Research  Approach 

Our  proposed  strategy  for  identifying  trends  in  SSSTRP  findings  on  weapon-system- 
related  acquisition  programs  is  to  initially  meet  with  PMs,  US  Navy  safety  community  members, 
and  system  developers  to  discuss  experiences  and  lessons  learned.  We  can  also  gather  data 
from  informal  meetings  and  discussions  and  analyze  it  to  find  more  quantitative  sources  of 
information. 

In  addition  to  collecting  data  from  various  participants  from  the  acquisition  community, 
we  will  also  request  and  analyze  TDPs,  SSSTRP  reports,  and  WSESRB  data  related  to  recent 
navy  gun  system  acquisition  programs.  We  expect  this  information  to  yield  quantitative  data  that 
can  assist  us  as  we  identify  trends  across  different  acquisition  programs.  Due  to  the  nature  of 
the  OA  initiative,  we  should  be  able  to  derive  trends  specific  to  OA  software-intensive  systems 
from  the  data,  as  well. 

The  third  step  in  this  research  approach  is  to  analyze  the  SSSTRP  process  itself  with  the 
goal  of  quantitatively  determining  the  consistency  of  the  process.  This  information  is  a  vital 
data-collection  objective  because  specifics  in  the  consistency  of  the  SSSTRP  process  from 
system  to  system  should  correlate  with  similar  results  for  similar  cases  being  presented  to  the 
panel. 


Our  fourth  step  in  this  research  approach  is  to  connect  the  identified  lessons  learned  and 
trends  in  SSSTRP  data  reports  with  data  on  the  SSSTRP  process  itself.  This  step  is  essentially 
the  point  at  which  we  can  begin  to  derive  conclusions  and  recommendations.  Relationships  in 
the  data  will  primarily  focus  on  how  to  most  effectively  reduce  programmatic  risk  in  guiding  a 
weapon  system  acquisition  program  through  the  SSSTRP  process  smoothly.  In  this 
assessment,  we  will  pay  particular  attention  to  system  characteristics  that  will  be  consistent 
throughout  future  programs,  such  as  OA  software. 
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The  end  result  of  this  research  will  be  to  create  a  deliverable  that  can  be  used  by  PMs  to 
more  effectively  manage  their  OA-based  acquisition  system  programs.  A  side  deliverable  will 
be  an  analysis  of  the  SSSTRP  process  and  recommendations  on  how  personnel  involved  in  the 
safety  community  can  improve  it. 


Current  Research  Status 

Data  collection  has  been  a  time-consuming  task,  but  progress  is  ongoing.  From  October 
to  December  2008,  we  conducted  both  project-scope  refinement  and  meetings  with  safety  and 
acquisition  community  members.  We  established  relationships  with  the  Naval  Gunnery  Project 
Office,  NSWC  Port  Hueneme  Division  (PHD)  Detachment  Louisville,  NSWC  Dahlgren  and 
Naval  Ordnance  Safety  and  Security  Activity  (NOSSA).  We  also  discussed  specifics  of  the 
process  with  a  majority  of  players  in  the  community  and  compiled  information  on  the  process 
itself.  Any  stakeholders  we  have  missed  are  encouraged  to  contact  us. 

Along  with  SSSTRP  and  WSESRB  process  data,  we  received  recent  gun-system-related 
SSSTRP  reports  through  a  request  from  the  Naval  Gunnery  Project  Office  to  NSWC  Dahlgren. 
For  research  purposes,  we  also  have  had  access  to  reports  deemed  releasable  by  the  Navy 
from  the  past  six  years.  TDPs  were  not  made  available  for  more  detailed  analysis  due  to  the 
sensitivity  of  the  majority  of  the  programs.  The  above-mentioned  data  were  received  in 
February  2009,  and  our  primary  focus  has  been  to  determine  what  information  is  most  useful  for 
analysis.  Collaboration  has  taken  place  with  NOSSA  personnel  to  determine  how  to  break 
down  and  classify  findings  from  the  SSSTRP  reports.  The  analysis  is  ongoing,  but  research  is 
still  in  the  early  stages  at  this  time. 
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Planning  Processes 

■  Managing  Services  Supply  Chain 

■  MOSA  Contracting  Implications 
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Contract  Management 
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■  Joint  Contingency  Contracting 

■  Model  for  Optimizing  Contingency  Contracting  Planning  and  Execution 

■  Navy  Contract  Writing  Guide 
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■  Lessons  from  Private  Sector  Capital  Budgeting  for  DoD  Acquisition  Budgeting 
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■  PPPs  and  Government  Financing 

■  ROI  of  Information  Warfare  Systems 

■  Special  Termination  Liability  in  MDAPs 

■  Strategic  Sourcing 

■  Transaction  Cost  Economics  (TCE)  to  Improve  Cost  Estimates 
Human  Resources 

■  Indefinite  Reenlistment 

■  Individual  Augmentation 
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■  Moral  Conduct  Waivers  and  First-tern  Attrition 

■  Retention 

■  The  Navy’s  Selective  Reenlistment  Bonus  (SRB)  Management  System 

■  Tuition  Assistance 
Logistics  Management 
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■  Cold-chain  Logistics 

■  Contractors  Supporting  Military  Operations 
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■  PBL  (4) 

■  Privatization-NOSL/NAWCI 

■  RFID  (6) 
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■  Sense-and-Respond  Logistics  Network 

■  Strategic  Sourcing 
Program  Management 

■  Building  Collaborative  Capacity 

■  Business  Process  Reengineering  (BPR)  for  LCS  Mission  Module  Acquisition 

■  Collaborative  IT  Tools  Leveraging  Competence 

■  Contractor  vs.  Organic  Support 
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■  KVA  Applied  to  Aegis  and  SSDS 

■  Managing  the  Service  Supply  Chain 

■  Measuring  Uncertainty  in  Earned  Value 

■  Organizational  Modeling  and  Simulation 

■  Public-Private  Partnership 

■  Terminating  Your  Own  Program 

■  Utilizing  Collaborative  and  Three-dimensional  Imaging  Technology 

A  complete  listing  and  electronic  copies  of  published  research  are  available  on  our  website: 
www.acquisitionresearch.org 


DEFENSE  ACQUISITION  IN  TRANSITION 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


JSSL 

ru  ku \tiiM 


DEFENSE  ACQUISITION  IN  TRANSITION 


ACQUISITION  RESEARCH  PROGRAM 

GRADUATE  SCHOOL  OF  BUSINESS  &  PUBLIC  POLICY 

NAVAL  POSTGRADUATE  SCHOOL 

555  DYER  ROAD,  INGERSOLL  HALL 

MONTEREY,  CALIFORNIA  93943 


www.acquisitionresearch.org 


1  9  0 


Defense  Acquisition  in  Transition 

6th  Annual  Acquisition  Research  Symposium 

—  m-Mn 


Software  Safety  Strategy  for  US  Navy 
Gun  System  Acquisition  Programs 


Joey  Rivera,  Paul  Dailey 

Software  Engineering  Ph.D.  Students,  Naval  Postgraduate  School 


Background 


•  US  Navy  weapon  and  combat  system 
acquisition  programs  are  evolving 

-  Drive  to  implement  Open  Architecture  (OA) 

-  Increasing  levels  of  software  complexity 


•  Acquisition  and  demonstration  of  safe 
software  becomes  a  more  challenging  task 

-  Increased  safety  risk,  cost  &  schedule  overruns 

-  Difficulty  proving  system  safety  to  independent 
review  boards 
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Problem 


•  There  is  no  standard  methodology  in  place 
for  program  managers  to  follow  regarding 
software  safety 

-  Current  management  practice  varies  from 
program  to  program 

-  Reactive  vs.  proactive  evaluation  approach 


-  Focus  of  this  research:  gun  systems  software 
safety 
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Software  Safety  Program  Management 
Strategy  Needed 


•  Methodical  and  effective  approach 

•  Strategy  goals 


-  Reduce  average  number  of  safety  issues 

-  Improve  process  for  handling  issues 
encountered 

-  Reduce  surprises  encountered  during 
SSSTRP  and  WSESRB 
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Developing  the  Risk  Management 
Strategy 

•  Identify  common  risks... 

-  Among  current  gun  system  acquisition  programs 

-  For  future  OA  software-based  gun  systems 


•  Develop  mitigation  strategies  to  address  each 
common  risk 


•  Combine  into  a  program  management  level 
software  safety  risk  management  strategy 
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Identifying  Common  Safety  Risks  for 
Today’s  Gun  Systems 

•  Conduct  survey,  discuss  experiences  & 
lessons  learned 

-  Program  Managers  &  Safety  Community  Members 


•  Analyze  SSSTRP  process: 


-  Panel  members 

-  Characteristics  of  systems  being  reviewed 


•  Research  OA  /  COTS  Specific  Risks 
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Identifying  Common  Safety  Risks  for 
Today’s  Gun  Systems  (cont) 

•  Obtain  SSSTRP  reports  on  recent  gun 
system  acquisition  programs 

•  Extract  &  catalog  findings  from  each  report 

-  Categorize  findings  into  project  management  and 
safety  management  areas 


•  Analyze  data,  identifying  common  risks  & 
trends 

-  Identify  OA-related  issues 
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Organize  Findings  from  SSSTRP  data 


•  Project  Management 


Safety  Management 


-  Project  Planning 

-  Requirements 
Management 

-  Integration  &  Testing 

-  Configuration 
Management 

-  Validation  &  Verification 


-  System  Safety  Program 

-  Software  Safety  Program 

-  Safety  Risk  Management 

-  Safety  Verification  / 
Audits 

-  Hazard  Tracking 

-  COTS,  GOTS,  NDI 


Risk  Management 

Deployment  & 
Maintenance 


-  Sim,  Stim,  Emulation 

Category  definitions  are  evolving  via 
collaboration  with  various  members  of  the 
DoD  systems  safety  community 
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Developing  Risk  Mitigation  Strategies 


Identify  successful  actions  used  to  resolve 
historical  issues 

Apply  existing/proven  risk  mitigation 
methodologies  from  OA  and  PM  domains 

Develop  custom  techniques  if  needed 

Continue  a  centralized  SSSTRP  findings 
database  to  track  future  opportunities 


Developing  Risk  Mitigation  Strategies 
(cont) 


•  Combine  risk  mitigation  methodologies  and 
techniques  into  a  program  management  methodology 

•  Provide  methodology  and  assessments  of  the 
content  to  program  managers  for  review  and  use 

•  Acquire  feedback  if  possible  to  improve  methodology 
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Software  Safety  Strategy  Development 


Data  Collection  in  Progress 


Filename  N3C5-G-10-146-1912.PDF 
MK  34  MOD  4  -  USS  Bunker  Hill  (CG-52) 
Purpose: 


Concurrance  to  conduct  structural  test  firings. 

Finding  Summary: 

•  Incorporate  accurate  data  into  software  test  build.  (Insufficient  Testing) 

•  Perform  safety  assessment  after  the  modifications  have  been  made. 

•  Verify  and  Validate  software  before  structural  testing 

•  Incorporate  safety  schedule  into  program  schedule 

•  Perform  interface  safety  assessment  between  training  system  and  weapon. 

•  Show  that  all  safety  risks  have  been  accepted  in  accordance  to  DoDI  500.2 

•  Establish  a  Hazard  Tracking  Database 

•  Provide  status  of  prior  SSSTRP  Findings. 


Example  of  data  extracted  from 
an  SSSTRP  report  for  current 
gun  programs 


Comments: 


The  SSSTRP  took  exception  to  the  fact  that  this  program  decided  to  seek  concurrance  to  conduct  structural 
test  firings  with  software  that  was  still  under  development.  I  suspect  that  the  previous  version  of  this 
software  had  been  accepted  by  the  SSSTRP  but  safety  assessments  need  to  be  performed  and  presented 
to  the  SSSTRP.  Also,  the  risks  associated  with  the  changes  needs  to  be  determined. 


Several  years  of  historical  SSSTRP  data  under 
analysis  so  far 
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Questions? 
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